\clearpage

```{r include = FALSE}

require(tidyverse)
require(scales)
require(knitr)
require(kableExtra)

load("data/botornot_prediction.RData")

```

# Authentic vs inauthentic coordination

To assess if there is any indication an inauthentic coordination between the far-right accounts, and in addition to the bot detection algorithm, we compute three different set of measurements. We are first interested in detecting possible evidence of programmatic use of multiple Twitter accounts. These measurements are designed to identify traces of specific forms of programmatic coordination but can't be used to exclude the presence of other forms of inauthentic coordination. Second, we report on manually coding a sample of 50 far-right accounts.  

## Bot detection

We analysed the users in our dataset to verify whether they were socialbots. Socialbots are computer-controlled online accounts that aim to appear as authentic as possible to the rest of the social media users, in order to create the illusion of popularity or critical mass (Gehl & Bakardjieva, 2016). We detected socialbots using the tweetbotornot2 R package (Ferrara et al., 2016), which used more than 1000 features to estimate the probability of an account being a social bots only based on our dataset (since a number of accounts were not active anymore, using live data was not an option). These features included: when the account was created, the number of followers, the levels of activity and whether a tweet was published using the Twitter web client or a mobile app. According to the software, 0.4% of accounts active during the Bushfire conversation and 0.7% during the Covid-10 conversation were assigned a likelihood of being a bot higher than 80%. Such a low percentage can be explained by Twitter’s own intervention between data creation and data collection.

Among the `r length(farright_user_ids)` far-right accounts, no account was assigned a likelihood of being a bot above 80%. In Period 1, 95% of the far-right accounts where assigned a value below `r round(quantile(bushfire_botornot$prob_bot[bushfire_botornot$user_id %in% farright_user_ids], p = .95)*100, 4)`% while in Period 2 a value below `r round(quantile(covid_botornot$prob_bot[covid_botornot$user_id %in% farright_user_ids], p = .95)*100, 4)`%. 

```{r results = 'asis'}


bushfire_botornot %>%
  dplyr::filter(user_id %in% farright_user_ids) %>%
  dplyr::mutate(`Period 1 %` = round(prob_bot * 100, 2)) %>%
  dplyr::full_join(covid_botornot %>%
                   dplyr::filter(user_id %in% farright_user_ids) %>%
                   dplyr::mutate(`Period 2 %` = round(prob_bot * 100, 2)),
                   by = "user_id") %>%
  dplyr::select(`Period 1 %`, `Period 2 %`) %>%
  dplyr::arrange(desc(`Period 1 %`), desc(`Period 2 %`)) %>%
  kableExtra::kbl(booktabs = T, longtable = T, 
                  caption = "Estimated likelihood of being a bot for the far-right accounts (n=208)") %>%
  kableExtra::kable_styling(latex_options = c("striped", "scale_down", "repeat_header"),
                            font_size = 8)


```

In the figure below, we first (left-panel) compare the mean and median of the estimated probabilities of accounts retweeting tweets from four categories of accounts: far-right, journalists, politicians and others. Then (right-panel) we compare the percentage of accounts retweeting the same categories with an estimated probability of being a bot of more than 80%. Overall, we notice that the presence of estimated bots (with a probability of 80% or more) have increased in Period 2 by more than three times over Period 1. But we also observe that far-right accounts seem to be quite surprisingly the category of accounts that is less retweeted by suspected bots. In Period 2, a regular account is estimated to be retweeted by a bot almost 19 times more often than a far-right account while a journalist almost 9 times more often than a far-right account.


```{r bot_rweets, include = T, fig.width = 7, fig.height = 3, fig.cap = "Percentage of retweets from accounts with an high probability of being bots (>80 perc.)"}

bot_retweet_stats.df <- 
  rbind(
    bushfire_tweets.dt %>%
      dplyr::filter(!is.na(retweeted_status_user_id)) %>%
      dplyr::mutate(cat =
                      case_when(
                        retweeted_status_user_id %in% 
                          farright_user_ids ~ "far-right retweet",
                        retweeted_status_user_id %in% 
                          journalist_user_ids ~ "journalist retweet",
                        retweeted_status_user_id %in% 
                          politician_user_ids ~ "politician retweet",
                        !retweeted_status_user_id %in% 
                          c(farright_user_ids,
                            journalist_user_ids,
                            politician_user_ids) ~ "other retweet"
                        
                      ),
                    bot_prob = 
                      bushfire_botornot$prob_bot[match(user_id,
                                                       bushfire_botornot$user_id)]) %>%
      dplyr::group_by(cat) %>%
      dplyr::summarise(median = median(bot_prob),
                       mean = mean(bot_prob),
                       "over 80% prob." = sum(bot_prob >= .8) / n()) %>%
      dplyr::mutate(when = "Bushfires"),
    
    covid_tweets.dt %>%
      dplyr::filter(!is.na(retweeted_status_user_id)) %>%
      dplyr::mutate(cat =
                      case_when(
                        retweeted_status_user_id %in% 
                          farright_user_ids ~ "far-right retweet",
                        retweeted_status_user_id %in% 
                          journalist_user_ids ~ "journalist retweet",
                        retweeted_status_user_id %in% 
                          politician_user_ids ~ "politician retweet",
                        !retweeted_status_user_id %in% 
                          c(farright_user_ids,
                            journalist_user_ids,
                            politician_user_ids) ~ "other retweet"
                        
                      ),
                    bot_prob = 
                      covid_botornot$prob_bot[match(user_id,
                                                    covid_botornot$user_id)]) %>%
      dplyr::group_by(cat) %>%
      dplyr::summarise(median = median(bot_prob),
                       mean = mean(bot_prob),
                       "over 80% prob." = sum(bot_prob >= .8) / n()) %>%
      dplyr::mutate(when = "Covid-19")) %>%
  dplyr::mutate(cat = factor(cat, levels = c("far-right retweet",
                                             "journalist retweet",
                                             "politician retweet",
                                             "other retweet")))

cowplot::plot_grid(
  bot_retweet_stats.df %>%
    dplyr::select(-`over 80% prob.`) %>%
    tidyr::pivot_longer(median:mean) %>%
    ggplot(aes(x = name, y = value, fill = cat)) +
    geom_bar(stat = 'identity', position = 'dodge') +
    scale_y_continuous(labels = scales::percent) +
    scale_fill_brewer(palette = "Set3") +
    theme_bw() + labs(x = NULL, y = NULL) + guides(fill = FALSE) +
    facet_wrap("when"),
  bot_retweet_stats.df %>%
    dplyr::select(-median, -mean) %>%
    tidyr::pivot_longer(`over 80% prob.`) %>%
    ggplot(aes(x = name, y = value, fill = cat)) +
    geom_bar(stat = 'identity', position = 'dodge') +
    scale_y_continuous(labels = scales::percent) +
    scale_fill_brewer(palette = "Set3") +
    theme_bw() + labs(x = NULL, y = NULL, fill = NULL) + 
    theme(legend.position = 'bottom', 
          legend.direction ="vertical") +
    facet_wrap("when"),
  ncol = 2, rel_widths = c(1, .8))

```

Our Twitter data was not collected in real-time but a few months after the events. Contrary to Facebook's policy, creating a Twitter bot is not against the terms of service. In fact, Twitter encourage some level of automation by giving access to an API to control all the posting functions of an account. And yet, “coordinated activity, that attempts to artificially influence conversations through the use of multiple accounts, fake accounts, automation and/or scripting” are against Twitter rules. It is then possible that a number of bots were removed from the data before our data was collected.  But it is also possible that bots retweeting far-right tweets were also more likely to be suspended by Twitter.


## Programmatic coordination

To understand if there is any indication of programmatic coordination to promote specific tweets through a coordinated retweet action we measure the difference in timestamps in the far-right retweet network and in the global retweet network.  

### Programmatic retweet coordination: Far-right retweet network

We define this network as the network of far-right accounts retweeting the same content. For each pair of far-right accounts connected by a common retweet, we measure the difference in seconds between the two retweets. We would expected a programmatic coordinated use of multiple accounts to retweets the same content within close interval to simulate a high-level of genuine engagement around the content. For each period we observe:

1. The size of the far-right retweet network (number of nodes); 
2. The number of retweets by the far-right accounts;
3. The number of ties of the retweet network;
4. The number of simultaneous retweets by far-right accounts (0 second difference);
5. The number of near-simultaneous retweets by far-right accounts (10 or less seconds difference).
6. The distribution of difference in seconds for all the retweets in the far-right network compared to the same distribution measured from a retweet network of 10,000 random accounts. 

#### Period 1

```{r}

# Far-right retweet network

bushfire_farright_retweet.el <- 
  bushfire_tweets.dt %>%
  dplyr::filter(user_id %in% 
                  coded_users$user_id[coded_users$far_right] & 
                  !is.na(retweeted_status_id)) %>%
  dplyr::select(tweet_id, created_at, user_id, retweeted_status_id)

bushfire_farright_retweet_joined.el <- 
  bushfire_farright_retweet.el %>%
  dplyr::full_join(bushfire_farright_retweet.el, by = "retweeted_status_id") %>%
  dplyr::mutate(same = tweet_id.x == tweet_id.y,
                diff = difftime(created_at.x, created_at.y, units = "secs")) %>%
  dplyr::filter(!same & diff >= 0) 

# Random retweet network

bushfire_random_retweet.el <- 
  bushfire_tweets.dt %>%
  dplyr::filter(!is.na(retweeted_status_id)) %>%
  dplyr::sample_n(100000) %>%
  dplyr::select(tweet_id, created_at, user_id, retweeted_status_id)

bushfire_random_retweet_joined.el <- 
  bushfire_random_retweet.el %>%
  dplyr::full_join(bushfire_random_retweet.el, by = "retweeted_status_id") %>%
  dplyr::mutate(same = tweet_id.x == tweet_id.y,
                diff = difftime(created_at.x, created_at.y, units = "secs")) %>%
  dplyr::filter(!same & diff >= 0) 

```

1. The size of the far-right retweet network: `r length(unique(bushfire_farright_retweet.el$user_id))`.

2. Number of retweets by far-right accounts: `r nrow(bushfire_farright_retweet.el)`.

3. Number of ties of the retweet network: `r nrow(bushfire_farright_retweet_joined.el)`

4. Number of simultaneous retweets by far-right accounts: `r sum(as.numeric(bushfire_farright_retweet_joined.el$diff) == 0)`

5. Number of near-simultaneous retweets by far-right accounts: `r sum(as.numeric(bushfire_farright_retweet_joined.el$diff) <= 10)` (`r round((sum(as.numeric(bushfire_farright_retweet_joined.el$diff) <= 10) /
        nrow(bushfire_farright_retweet_joined.el)) * 100, 4)`)
        
```{r include = F, results = 'asis', fig.cap = 'Distribution of difference in seconds in far-right network compared to random accounts (Period 1)'}

bushfire_farright_retweet_joined.el %>%
  dplyr::mutate(what = "far-right accounts",
                diff = as.numeric(diff)) %>%
  bind_rows(bushfire_random_retweet_joined.el %>%
              dplyr::mutate(what = "random accounts",
                            diff = as.numeric(diff))) %>%
  ggplot(aes(x = diff, colour = what)) +
  geom_density() +
  scale_x_continuous(trans=scales::pseudo_log_trans(base = 10),
                     breaks = c(10, 100, 1000, 10000, 100000, 1000000),
                     labels = comma) +
  labs(x = "Difference between retweets in seconds", colour = NULL) +
  theme_bw()

```


#### Period 2

```{r}

# Far-right retweet network

covid_farright_retweet.el <- 
  covid_tweets.dt %>%
  dplyr::filter(user_id %in% 
                  coded_users$user_id[coded_users$far_right] & 
                  !is.na(retweeted_status_id)) %>%
  dplyr::select(tweet_id, created_at, user_id, retweeted_status_id)

covid_farright_retweet_joined.el <- 
  covid_farright_retweet.el %>%
  dplyr::full_join(covid_farright_retweet.el, by = "retweeted_status_id") %>%
  dplyr::mutate(same = tweet_id.x == tweet_id.y,
                diff = difftime(created_at.x, created_at.y, units = "secs")) %>%
  dplyr::filter(!same & diff >= 0) 

# Random retweet network

covid_random_retweet.el <- 
  covid_tweets.dt %>%
  dplyr::filter(!is.na(retweeted_status_id)) %>%
  dplyr::sample_n(100000) %>%
  dplyr::select(tweet_id, created_at, user_id, retweeted_status_id)

covid_random_retweet_joined.el <- 
  covid_random_retweet.el %>%
  dplyr::full_join(covid_random_retweet.el, by = "retweeted_status_id") %>%
  dplyr::mutate(same = tweet_id.x == tweet_id.y,
                diff = difftime(created_at.x, created_at.y, units = "secs")) %>%
  dplyr::filter(!same & diff >= 0) 

```

1. The size of the far-right retweet network: `r length(unique(covid_farright_retweet.el$user_id))`.

2. Number of retweets by far-right accounts: `r nrow(covid_farright_retweet.el)`.

3. Number of ties of the retweet network: `r nrow(covid_farright_retweet_joined.el)`

4. Number of simultaneous retweets by far-right accounts: `r sum(as.numeric(covid_farright_retweet_joined.el$diff) == 0)`

5. Number of near-simultaneous retweets by far-right accounts: `r sum(as.numeric(covid_farright_retweet_joined.el$diff) <= 10)` (`r round((sum(as.numeric(covid_farright_retweet_joined.el$diff) <= 10) /
        nrow(covid_farright_retweet_joined.el)) * 100, 4)`)
        
```{r include = F, results = 'asis', fig.cap = 'Distribution of difference in seconds in far-right network compared to random accounts (Period 2)'}

covid_farright_retweet_joined.el %>%
  dplyr::mutate(what = "far-right accounts",
                diff = as.numeric(diff)) %>%
  bind_rows(covid_random_retweet_joined.el %>%
              dplyr::mutate(what = "random accounts",
                            diff = as.numeric(diff))) %>%
  ggplot(aes(x = diff, colour = what)) +
  geom_density() +
  scale_x_continuous(trans=scales::pseudo_log_trans(base = 10),
                     breaks = c(10, 100, 1000, 10000, 100000, 1000000),
                     labels = comma) +
  labs(x = "Difference between retweets in seconds", colour = NA) +
  theme_bw()

```


### Programmatic retweet coordination: Global retweet network

We measure the difference in seconds between reweets of far-right accounts made by any account. We would expect programmatic coordination to boost the engagement around a tweet to be reflected by relatively short time differences between retweets of far-right content. We then compare the distribution of differences for retweets of far-right content to the distribution measured for retweets of random accounts. 

For each period, we observe:

1. The number of retweets of far-right tweets. 
2. The distribution of differences in seconds between the first and the second retweet. 
3. The distribution of differences in seconds between the time of the first retweet and the time of the first 50\% of retweets.

#### Period 1

```{r}

bushfire_retweeted.dt <- 
  bushfire_tweets.dt %>%
  dplyr::mutate(farright = retweeted_status_user_id %in% 
                  coded_users$user_id[coded_users$far_right]) %>%
  dplyr::select(tweet_id, created_at, user_id, retweeted_status_id, farright) %>%
  dplyr::filter(!is.na(retweeted_status_id))

samples <- 
  list(sample(unique(bushfire_retweeted.dt$retweeted_status_id[!bushfire_retweeted.dt$farright]),
              10000),
       unique(bushfire_retweeted.dt$retweeted_status_id[bushfire_retweeted.dt$farright])
  )

bushfire_retweeted.dt <- 
  bushfire_retweeted.dt %>%
  dplyr::filter(retweeted_status_id %in% unlist(samples)) %>%
  dplyr::group_by(retweeted_status_id, farright) %>%
  dplyr::summarize(
    timediff_1st_2nd =
      as.numeric(difftime(sort(created_at)[2],
                          sort(created_at)[1],
                          units = 'secs')),
    timediff_1st_50perc =
      as.numeric(difftime(sort(created_at)[length(created_at) %/% 2],
                          sort(created_at)[1],
                          units = 'secs')),
    n = n())

```

1. The number of retweets of far-right tweets: `r nrow(bushfire_retweeted.dt)`

```{r echo = F, results = 'asis'}

bushfire_retweeted.dt %>%
  dplyr::filter(n >= 10) %>%
  ggplot() +
  geom_density(aes(x = timediff_1st_2nd,
                   colour = farright)) +
  scale_x_continuous(trans=scales::pseudo_log_trans(base = 10),
                     breaks = c(10, 100, 1000, 10000, 100000, 1000000),
                     labels = comma) +
  theme_bw() +
  labs(x = "Difference in seconds between 1st and 2nd retweet", 
       colour = "far-right")

```

```{r, echo = F, results = 'asis'}

bushfire_retweeted.dt %>%
  dplyr::filter(n >= 10) %>%
  ggplot() +
  geom_density(aes(x = timediff_1st_50perc,
                   colour = farright)) +
  scale_x_continuous(trans=scales::pseudo_log_trans(base = 10),
                     breaks = c(10, 100, 1000, 10000, 100000, 1000000),
                     labels = comma) +
  theme_bw() +
  labs(x = "Difference in seconds between 1st retweet and 1st 50% of retweets", 
       colour = "far-right")

```

#### Period 2

```{r}

covid_retweeted.dt <- 
  covid_tweets.dt %>%
  dplyr::mutate(farright = retweeted_status_user_id %in% 
                  coded_users$user_id[coded_users$far_right]) %>%
  dplyr::select(tweet_id, created_at, user_id, retweeted_status_id, farright) %>%
  dplyr::filter(!is.na(retweeted_status_id))

samples <- 
  list(sample(unique(covid_retweeted.dt$retweeted_status_id[!covid_retweeted.dt$farright]),
              10000),
       unique(covid_retweeted.dt$retweeted_status_id[covid_retweeted.dt$farright])
  )

covid_retweeted.dt <- 
  covid_retweeted.dt %>%
  dplyr::filter(retweeted_status_id %in% unlist(samples)) %>%
  dplyr::group_by(retweeted_status_id, farright) %>%
  dplyr::summarize(
    timediff_1st_2nd =
      as.numeric(difftime(sort(created_at)[2],
                          sort(created_at)[1],
                          units = 'secs')),
    timediff_1st_50perc =
      as.numeric(difftime(sort(created_at)[length(created_at) %/% 2],
                          sort(created_at)[1],
                          units = 'secs')),
    n = n())

```

1. The number of retweets of far-right tweets: `r nrow(covid_retweeted.dt)`

```{r echo = F, results = 'asis'}

covid_retweeted.dt %>%
  dplyr::filter(n >= 10) %>%
  ggplot() +
  geom_density(aes(x = timediff_1st_2nd,
                   colour = farright)) +
  scale_x_continuous(trans=scales::pseudo_log_trans(base = 10),
                     breaks = c(10, 100, 1000, 10000, 100000, 1000000),
                     labels = comma) +
  theme_bw() +
  labs(x = "Difference in seconds between 1st and 2nd retweet", 
       colour = "far-right")

```

```{r, echo = F, results = 'asis'}

covid_retweeted.dt %>%
  dplyr::filter(n >= 10) %>%
  ggplot() +
  geom_density(aes(x = timediff_1st_50perc,
                   colour = farright)) +
  scale_x_continuous(trans=scales::pseudo_log_trans(base = 10),
                     breaks = c(10, 100, 1000, 10000, 100000, 1000000),
                     labels = comma) +
  theme_bw() +
  labs(x = "Difference in seconds between 1st retweet and 1st 50% of retweets", 
       colour = "far-right")

```

## Programmatic tweet coordination: Same text coordination

We calculate the number of tweets containing the exact same text published by far-right accounts. We observe, 

1. How many times same-text messages were posted by far-right accounts; 
2. How many unique same-text messages were posted by far-right accounts

### Period 1

```{r}

bushfire_farright_tweets.dt <-
  bushfire_tweets.dt %>%
  dplyr::filter(user_id %in% 
                  coded_users$user_id[coded_users$far_right] & 
                  is.na(retweeted_status_id)) %>%
  dplyr::select(tweet_id, created_at, user_id, text)

bushfire_farright_same_text_tweets.dt <- 
  bushfire_farright_tweets.dt %>%
  dplyr::full_join(bushfire_farright_tweets.dt, by = 'text') %>%
  dplyr::mutate(same = tweet_id.x == tweet_id.y,
                diff = difftime(created_at.x, created_at.y, units = "secs")) %>%
  dplyr::filter(!same & diff >= 0) 

```

1. Same-text messages were posted by far-right accounts: `r sum(bushfire_farright_tweets.dt$tweet_id %in% c(bushfire_farright_same_text_tweets.dt$tweet_id.x,
        bushfire_farright_same_text_tweets.dt$tweet_id.y))` (`r round(sum(bushfire_farright_tweets.dt$tweet_id %in% c(bushfire_farright_same_text_tweets.dt$tweet_id.x,
        bushfire_farright_same_text_tweets.dt$tweet_id.y)) / 
  nrow(bushfire_farright_tweets.dt) * 100, 4)`\%).

2. Unique same-text messages were posted by far-right accounts: `r length(unique(bushfire_farright_same_text_tweets.dt$text))`.

### Period 2

```{r}

covid_farright_tweets.dt <-
  covid_tweets.dt %>%
  dplyr::filter(user_id %in% 
                  coded_users$user_id[coded_users$far_right] & 
                  is.na(retweeted_status_id)) %>%
  dplyr::select(tweet_id, created_at, user_id, text)

covid_farright_same_text_tweets.dt <- 
  covid_farright_tweets.dt %>%
  dplyr::full_join(covid_farright_tweets.dt, by = 'text') %>%
  dplyr::mutate(same = tweet_id.x == tweet_id.y,
                diff = difftime(created_at.x, created_at.y, units = "secs")) %>%
  dplyr::filter(!same & diff >= 0) 

```

1. Same-text messages were posted by far-right accounts: `r sum(covid_farright_tweets.dt$tweet_id %in% c(covid_farright_same_text_tweets.dt$tweet_id.x,
        covid_farright_same_text_tweets.dt$tweet_id.y))` (`r round(sum(covid_farright_tweets.dt$tweet_id %in% c(covid_farright_same_text_tweets.dt$tweet_id.x,
        covid_farright_same_text_tweets.dt$tweet_id.y)) / 
  nrow(covid_farright_tweets.dt) * 100, 4)`\%).

2. Unique same-text messages were posted by far-right accounts: `r length(unique(covid_farright_same_text_tweets.dt$text))`.

\clearpage

## Manual coding of far right accounts



```{r}

library(readr)

Far_right_manual_coding <- 
  read_csv("data/far-right-manual-coding.csv")

questions <- 
  names(Far_right_manual_coding)

questions <- 
  questions[3:9]

names(Far_right_manual_coding) <- 
  c("timestamp", 'user_id', 'q1', "q2", "q3", "q4", 
              "q5", "q6", "q7")

Far_right_manual_coding <- 
  Far_right_manual_coding %>%
  dplyr::mutate(q1_num = case_when(q1 == "Yes" ~ 1,
                                   q1 == "No" ~ 0),
                q2_num = case_when(q2 == "Yes" ~ 1,
                                   q2 == "No" ~ 0,
                                   q2 == "There's no `url` information" ~ NA_real_),
                q3_num = case_when(q3 == "Yes" ~ 0,
                                   q3 == "No" ~ 1),
                q4_num = case_when(q4 == "Yes" ~ 1,
                                   q4 == "No" ~ 0),
                q5_num = case_when(q5 == "Yes" ~ 1,
                                   q5 == "No" ~ 0,
                                   q5 == "Only one tweet available" ~ NA_real_),
                q6_num = case_when(q6 == "Yes" ~ 1,
                                   q6 == "No" ~ 0,
                                   q6 == "Account in not online anymore" ~ NA_real_),
                q7_num = case_when(q7 == "Yes" ~ 1,
                                   q7 == "No" ~ 0,
                                   q7 == "Account in not online anymore" ~ NA_real_))

Far_right_manual_coding <- 
  Far_right_manual_coding %>%
  dplyr::rowwise() %>%
  dplyr::mutate(`Answered %` = 
                  round(sum(!is.na(c(q1_num,
                                     q2_num,
                                     q3_num,
                                     q4_num,
                                     q5_num,
                                     q6_num,
                                     q7_num))) /
                          7 * 100, 
                        2),
                `Score %` = round(mean(c(q1_num,
                               q2_num,
                               q3_num,
                               q4_num,
                               q5_num,
                               q6_num,
                               q7_num), na.rm = T) * 100, 2),
                `Weigh. score %` = round(`Score %` / `Answered %` * 100, 2))
          
```

```{r include = T, results = "asis"}

data.frame(paste0("Q",1:7), 
           questions) %>%
  dplyr::bind_cols(Far_right_manual_coding %>%
                     group_by() %>%
                     summarise(across(q1_num:q7_num, ~ mean(.x, na.rm = TRUE))) %>%
                     t()) %>%
  dplyr::rename(`mean %` = `...3`) %>%
  dplyr::mutate(`mean %` = round(`mean %` * 100, 2)) %>%
  kbl(format = "latex", 
      caption = "Codebook for manual coding",
      booktabs = T,
      col.names = c("", "question", "mean (excluding NAs)")) %>%
  kable_styling(latex_options = c("striped", "repeat_header"), font_size = 8) %>%
  column_spec(1, width = "5em") %>%
  column_spec(2, width = "30em") %>%
  kable_styling(latex_options = "HOLD_position")

```

The inauthenticity scores are calculated summing the responses indicating potential inauthenticity and then dividing by the number of features in the codebook (seven) or the number of responses that could be answered (because information was available).  

* Mean inauthenticity score: `r round(mean(Far_right_manual_coding[["Score %"]]), 3)` %
* Mean inauthenticity score (weighted excluding NAs): `r round(mean(Far_right_manual_coding[["Weigh. score %"]]), 3)` %

```{r include = T, results = "asis"}

Far_right_manual_coding %>%
  dplyr::select(q1:q7, `Answered %`, `Score %`, `Weigh. score %`) %>%
  dplyr::arrange(desc(`Weigh. score %`)) %>%
  kbl(format = "latex", caption = "Results from manual coding a sample of 50 far-right accounts",
                       booktabs = T,
        longtable = T) %>%
  kable_styling(latex_options = c("striped", "repeat_header", "scale_down"), font_size = 8) %>%
  column_spec(1:7, width = "5em") %>%
  column_spec(8:10, width = "3em")
```


